1. Field of the Invention
The present invention relates to an image processing device, an image processing method, and a program, and particularly to an image processing device, an image processing method, and a program for performing super resolution processing that increases the resolution of an image.
2. Description of the Related Art
Super resolution (SR) is known as a technique for generating an image of high resolution from an image of low resolution. Super resolution (SR) generates an image of high resolution from an image of low resolution.
As a technique of super resolution processing, there is for example a reconstruction type super resolution technique that derives parameters indicating photographing conditions such as “blurs caused by a lens and atmospheric scattering,” “motion of a subject and a camera as a whole,” and “sampling by an image pickup element” on the basis of a photographed image of low resolution, and which estimates an ideal high-resolution image using these parameters.
Japanese Patent Laid-Open No. 2008-140012, for example, discloses a related-art technique in relation to a technique of the super resolution processing.
Outlines of the procedure of the reconstruction type super resolution technique are as follows.
(1) An image photographing model in which a blur, motion, sampling and the like are taken into account is expressed by a numerical formula.
(2) A cost calculating equation is obtained from the image photographing model expressed by the above numerical formula model. At this time, a regularization term of a priori probability or the like may be added using a Bayesian theory.
(3) An image minimizing the cost is obtained.
The reconstruction type super resolution technique obtains a super resolution image by these processes. While a high-resolution image obtained by the reconstruction type super resolution technique is dependent on an input image, a high degree of super resolution effect (resolution restoring effect) is obtained.
FIG. 1 shows an example of a circuit configuration for performing super resolution processing. FIG. 1 shows an example of a circuit configuration of a super resolution processing device 10.
In the image processing device 10, a low-resolution image gn 31 as an object of processing for increasing resolution is input to an upsampling section 11. The upsampling section 11 performs conversion of the number of pixels (image enlarging processing). Specifically, the upsampling section 11 performs the image enlarging processing that adjusts the number of pixels of the input image to the number of pixels of an image to be output (SR processed image fn 32), for example processing that divides one pixel into a plurality of pixels and which sets the plurality of pixels.
A motion estimated/motion compensated image generating section 12 detects the magnitude of a motion between a high-resolution image generated in the processing of a previous frame and the upsampled low-resolution image gn. Specifically, the motion estimated/motion compensated image generating section 12 calculates a motion vector. Further, using the detected motion vector, the motion estimated/motion compensated image generating section 12 performs motion compensation processing (MC) on the high-resolution image fn-1. A motion compensated image, which results from the motion compensation processing on the high-resolution image fn-1 and in which the position of a subject is set to be the same as in the upsampled low-resolution image gn, is thereby generated. However, when there is a moving subject or the like within the image, an image region in which the position of the subject is displaced from the upsampled low-resolution image gn, that is, a motion compensation failure region may occur in the motion compensated image.
A motion determining section 13 compares the motion compensated high-resolution image generated by the motion compensation (MC) processing and the upsampled low-resolution image with each other, and detects a region in which motion compensation cannot be applied well, that is, the above-described motion compensation failure region. The motion compensation failure region occurs in an image part where the subject itself is moving as described above, for example.
The motion determining section 13 generates motion region information (α-map [0:1]), which distinguishes a region in the motion compensated image of the high-resolution image fn-1 in which region the position of the subject is set to be the same as in the upsampled low-resolution image gn as a motion compensation success region and distinguishes a region in the motion compensated image of the high-resolution image fn-1 in which region the position of the subject is not set to be the same as in the upsampled low-resolution image gn as a motion compensation failure region. The motion determining section 13 then outputs the motion region information (α-map [0:1]). The motion region information (α-map [0:1]) is a map in which values in a range of one to zero are set according to reliability of success regions and failure regions. The motion region information (α-map [0:1]) can also be set simply as a map in which one is set as a motion compensation success region and zero is set as a motion compensation failure region, for example.
A blend processing section 14 is supplied with:
the motion compensation result image resulting from the motion compensation processing on the high-resolution image fn-1, the motion compensation result image being generated by the motion estimated/motion compensated image generating section 12;
the upsampled image obtained by upsampling the low-resolution image (gn) 31 in the upsampling section 11; and
the motion region information (α-map [0:1]).
Using these pieces of input information, the blend processing section 14 outputs a blended image as a blend result on the basis of the following equation.Blended Image=(1−α)(Upsampled Image)+α(Motion Compensation Result Image)
This blend processing generates a blended image in which the blend ratio of the motion compensation result image is raised for a motion compensation success region and the blend ratio of the motion compensation result image is lowered for a motion compensation failure region.
A blur adding section 15 is supplied with the blended image generated by the blend processing section 14, and performs simulation of degradation in spatial resolution. For example, the blur adding section 15 performs convolution into the image with a point spread function measured in advance as a filter.
A downsampling section 16 performs the processing of downsampling the high-resolution image to the same resolution as that of the input image. Thereafter, a difference unit 17 calculates the difference value of each pixel between the output image of the downsampling section 16 and the low-resolution image gn. The difference value is subjected to upsampling processing in an upsampling section 18. Further, an inverse blur adding section 19 performs processing reverse to the blur addition processing. As an operation, processing corresponding to calculation of a correlation with the PSF (Point Spread Function) used in the blur adding section 15 is performed.
The output of the inverse blur adding section 19 is multiplied by a feedback coefficient γ set in a multiplier 20 in advance, thereafter output to an adder 21 to be added to the blended image output by the blend processing section 14, and then output.
The output of the adder 21 is an image (SR processed image 32) obtained by converting the input image gn 31 to high resolution.
The processing performed by the super resolution processing device 10 shown in FIG. 1 can be expressed by an equation as follows.fnSR=(Wnfn-1SR)′+γSRHTDT(gn−DH(Wnfn-1SR)′)  (Equation 1)
Incidentally, each parameter in the above equation (Equation 1) is the following parameters.
n: frame number (an (n−1)th frame and an nth frame are for example consecutive frames of a moving image)
gn: input image (low-resolution image of the nth frame)
fnSR: super resolution processing result image (=high-resolution image) of the nth frame
fn-1SR: super resolution processing result image (=high-resolution image) of the (n−1)th frame
Wn: motion information (a motion vector, a matrix or the like) of the nth frame with respect to the (n−1)th frame
H: blur addition processing (blur filter matrix)
D: downsampling processing (downsampling processing matrix)
(Wnfn-1SR)′: the blended image output by the blend processing section
γSR: feedback coefficient
HT: the transposed matrix of H
DT: the transposed matrix of D
Problems in the case of constructing hardware for realizing the super resolution processing circuit 10 shown in FIG. 1 include an increase in circuit scale and a decrease in processing efficiency. For example, filter processing is applied to the processing of the upsampling section, the downsampling section, the blur adding section, and the inverse blur adding section shown in FIG. 1, and a RAM for storing pixel values is necessary to perform these many pieces of filtering processing. When the number of filtering taps is increased, a necessary memory capacity is also increased, and a high-performance CPU and the like are necessary to make high-speed memory access. There is thus a fear of an increase in hardware size and an increase in cost.